Virtual reality medical application system

ABSTRACT

Systems and methods are disclosed for monitoring a patient by positioning the patient for a predetermined medical mission; sensing biometric and physical conditions of a patient during the mission, and displaying a multimedia interaction with the patient to keep the patient in a predetermined position to improve efficacy of a medical mission.

RELATED APPLICATIONS

This application claims the benefit of priority to U.S. ProvisionalPatent Application Ser. No. 61/949,196 filed Mar. 6, 2014, the contentof which is fully incorporated herein incorporated by reference.

FIELD OF THE INVENTION

The present invention relates generally to medical treatment apparatus.The present invention is more particularly, though not exclusively,useful as a patient monitoring system that monitors both the biometricand physical condition of a patient, and provides a virtual realityinterface to that patient to improve the tolerance and efficacy of themedical treatment. The present invention is most particularly useful asa virtual reality environment which interfaces to medical treatmentapparatus and with medical service providers to decrease the discomfort,anxiety and pain of a patient during a procedure, such as radiationtherapy, MRI imaging, conoloscopy or biopsy.

BACKGROUND OF THE INVENTION

Radiation causes cellular degradation due to damage to DNA and other keymolecular structures within the cells in various tissues. It wasdiscovered that with controlled amounts of radiation, the effects causedby radiation can be utilized to treat certain diseases. With the use ofradiation, several previously untreatable diseases have been able to besuccessfully treated using radiation therapies. However, along with thecurative effects of radiation are negative side effects. The prolongedexposure to radiation include both immediate and delayed side effects.The immediate side effects of radiation exposure include nausea andvomiting, headaches, fevers, and skin burns, whereas the delayed effectscould include fatigue, weakness, hemorrhaging, leukopenia, epilation andother various side effects. Additionally, the prolonged overexposure ofunnecessary radiation may cause more serious complications and may leadto mortality. Due to the serious side effects of utilizing radiation asa form of medical treatment, numerous steps have been taken to limit theexposure of radiation to healthy tissue by focusing the radiation solelyon the diseased tissue.

External beam radiation therapy, the most common form of radiationtherapy, is the application of radiation at a particular part of thebody with the use of an external source of radiation. External beamradiation therapy directs concentrated radiation to the treatment areafrom outside of the body. To minimize the exposure of the concentratedradiation to healthy tissue, the concentrated radiation beam isprecisely controlled by the use of mechanical hardware and computersoftware. Current technology allows radiation to be precisely focused onthe diseased tissue occupying a minuscule area on a larger healthytissue. The use of multiple low-radiation beams focusing on an area ofdiseased tissue at different approach angles creates a single, focusedand concentrated radiation beam. This exposes the surrounding healthytissue to lower levels of radiation due to the use of multiplelow-radiation beams. The advancements in technology have allowed theprecise delivery of the concentrated beams to a diseased tissue whileminimizing the exposure to the surrounding healthy tissue.

The course of radiation treatment typically consists of multiple dailytreatments (20 to 40) over a course of 4 to 8 weeks, and each day theradiation treatment is given over a period of about 15-30 minutes.Although the directed radiation beam precisely delivers the radiation todiseased tissue, its frame of reference is to a stationary patient andtraditionally does not take into account the natural and unnaturalmovements of a patient during that 15-30 minute treatment. In moreadvanced techniques where higher levels of radiation are used, in orderto allow the patients to fully benefit from the procedure and minimizeits side effects, the delivery of the radiation must be precise,requiring radiation alignment accuracies in range of millimeter. Due tothe fact that the radiation dose is delivered over a period of 15-30minutes during which patient must remain still. The slight movement of apatient during a treatment will alter the point of delivery of theconcentrated radiation beam, therefore reducing the exposure ofradiation to the diseased tissue while increasing the exposure ofradiation to the healthy tissue. The treatments then become lesseffective which would require additional treatment, further exposing thesurrounding healthy tissue to unneeded radiation or other treatmenttypes. The patient's movements must be eliminated or minimized to ensureproper delivery of radiation to diseased tissue and minimize theexposure to healthy tissue.

Currently, there are two major ways to monitor movement of the patients.One is a technician must constantly watch the patient via a in-roomvideo to make sure they are not moving for up to a 30 minute period.This is error prone since it is subject to the technician interpretationof movement.

In order to minimize the movements, patients are traditionally subjectto invasive and uncomfortable methods of immobilization. Methods includeutilizing stabilizing frames attached to the body, stabilizing bodycasts or plastic molds, and internal spacers inserted within a patient'sbody to stabilize the diseased tissue. To control the tolerance of apatient's movement, tight clearance is required. This may causediscomfort and pain as the patient is tightly held in a single positionfor periods of up to 30 minutes for a single treatment. The physicalrestriction of the movements of the patient may cause or increase thelevels of anxiety, resulting in heavier and quicker breathing. This isparticularly important when treating diseased tissue that moves with anybodily function associated with breathing or involuntary cough. Forchildren with cancer that require radiation treatment and cannotcooperate to remain still, they require daily general anesthesia, wherethey have to be put to sleep with invasive sedative medications via anintravenous route and require oxygen through their nose and mouth. Thisdaily procedure typically costs an additional $5,000 to 6,000 per day,requires additional medical staff (such as anesthesiologists andnurses), medical equipment, and results in a longer treatment time.Besides being invasive, treatment involving general anesthesia requiresan additional 3-5 hours per day for these young patients due toinduction and recovery time.

Patients generally prefer to be unrestrained. In instances where the useof mechanical stabilization devices is undesirable or not feasible, thepatient must refrain from moving, including the momentary cessation ofbreathing. To ensure the patient has not moved from their initialposition and ensure proper alignment of the radiation with the diseasedtissue, a technician must continually monitor the movements of thepatient. The technician may be error prone as it requires thetechnician's interpretation of whether the patient has moved a distanceas little as few centimeters. Alternatively, the precise monitoring of apatient's bodily movement can be monitored using real-time 3-dimensionalsurface imaging techniques. Using real-time 3-dimensional surfaceimaging techniques, the patient's current body position is continuallycross-referenced with an ideal reference image, which is used tocalculate the coordinates to deliver radiation. If movement is detectedthe technician is alerted and the technician alerts the patient not tomove, eliminating the need for a technician's judgment of a patient'sbody position.

The current standard way to monitor movement of the patients requires atechnician to constantly watch the patient via an in-room video to makesure they are not moving for up to a 30 minute period. This is errorprone since it is subject to the technician interpretation of movement;it only shows only gross movement or body positioning. This system doesnot allow patient's participation and monitoring of his/her movement,and patients has no control or awareness. If the technician sees bodymovement, he/she has to re-enter to treatment room to re-adjust patientposition to match the reference position, and this adds additional timeto the treatment.

One system (the Varian® Real-time Position Management™ or RPM), acurrently existing medical device (VisionRT), uses real-time 3-D surfaceimaging incorporating an in-room camera to monitor movement. Thismonitoring is achieved by comparing the live three dimensional patientsurface images with reference images and the system provides thedifference between the 2 image sets to the technician. When movement isdetected and the workstation will alert the technician, the technicianalerts the patient not to move. There is some delay (deadtime) betweentime the movement is detected and the technician instruction to patient.This approach is even less effective when patients are experiencing painand uncontrolled movements, as well as those with hearing impairment orare simply too young to understand the technician's instructions.

The control of radiation exposure is particularly challenging forchildren because children are naturally more anxious than adults.Staying still for prolonged periods of times for the treatment is adifficult proposition for children and the use of mechanicalstabilization devices or sedatives is commonly used. The mechanicalstabilization devices generally cause additional anxiety in children asthey are bound and immobilized, resulting in crying, screaming, andminute movements as they struggle to become free. The use of sedatives,although effective to immobilize, comes with medical risk of thesedative medications, costs (up to several thousands dollars pertreatment), and extended induction and recovery time (3-5 hours pertreatment). This risk is repeated every day, 5 days per week, for periodof 3-6 weeks.

In light of the above, it would be advantageous to provide an apparatusand method to decrease the movement of patients undergoing radiotherapy.It would further be advantageous to provide an apparatus and method todecrease the movement of patients undergoing radiotherapy which does notuse physical restraints or chemical sedatives to immobilize a patient.It would further be advantageous to provide an apparatus and method todecrease the movement of patients undergoing radiotherapy which isnon-invasive and comfortable. It would further be advantageous toprovide an apparatus and method to decrease the movement of patientsundergoing radiotherapy which monitors the real-time position of apatient and relays the information to the patient about the specificarea of the body and allows patient to make adjustment to get back tothe reference or desired position. It would further be advantageous toprovide an apparatus and method to decrease the movement of patientsundergoing radiotherapy which alerts a third party (operator ortechnician) of excessive movement outside a set range and which part ofthe body. It would further be advantageous to provide an apparatus andmethod to assure the safely to patients undergoing radiotherapy whichsend an interrupt signal instantly to the radiation equipment to pausethe radiation beam due to excessive movement outside a set range. Itwould further be advantageous to provide an apparatus and method todecrease the movement of patients undergoing radiotherapy which providesa virtual reality system which visualizes the patient's current physicaland biometric measurements and allows for the real-time adjustment oftheir physical and biometric measurements in response.

SUMMARY OF THE INVENTION

The Virtual Reality Medical Application System of the present inventionis a medical system including a combination of hardware and softwarethat will improve a patient's ability to remain still during medicalprocedures that require a patient to not move. The system includes threeprimary components: the hardware component; the software component; anda sensor component. The hardware component consists of a threedimensional (3-D) goggle to provide visual images to patients,noise-cancellation headphone and microphone system to provide 3-waycommunication between the patient, the healthcare provider, and theVirtual Reality Medical Application System software.

The software component provides a variety of patient specific 3-D gamesthat are controlled by patients, virtual simulation of real life scenery(car racing, hot-air balloon, fish-eye view, etc), and may include realtime images of the procedure that a healthcare provider wants to sharewith patient, such as an inside image of the colon from the colonoscopy.The 3-D games and virtual simulation software will be developed is, in apreferred embodiment, run on linin, and the technician console is across platform and runs on Windows, Mac OSX, and Linux in a custommedical-grade computer with input/output ports and storage system.

The sensor component performs two major functions: motion tracking andbiofeedback. For motion tracking of the patients, sensors such asgyroscope and accelerometer sensors on the 3-D goggles to detect headmotion and one or more video cameras monitor patient body motions. Forbiofeedback, sensors such as blood pressure, heart rate, EEG, and EKG,among others will help a technician keep informed of the patientcondition, and will tend to determine the game play mechanics.

Alternatively, motion may be detected using a motion sensing device,capable of three-dimensional motion detection. One such sensor iscommercially available as the Kinect sensor. Kinect builds on softwaretechnology developed internally by Rare, a subsidiary of Microsoft GameStudios owned by Microsoft, and on range camera technology by Israelideveloper PrimeSense, which developed a system that can interpretspecific gestures, making completely hands-free control of electronicdevices possible by using an infrared projector and camera and a specialmicrochip to track the movement of objects and individuals in threedimensions. This 3D scanner system, often called Light Coding, employs avariant of image-based 3D reconstruction.

The Kinect sensor is a horizontal bar connected to a small base with amotorized pivot and is designed to be positioned lengthwise above orbelow the video display. The device features an “RGB camera, depthsensor and multi-array microphone running proprietary software” whichprovide full-body 3D motion capture, facial recognition and voicerecognition capabilities. Kinect sensor's microphone array enables itsattached devices, such as an Xbox 360, to conduct acoustic sourcelocalization and ambient noise suppression, allowing for things such asheadset-free party chat over Xbox Live.

The depth sensor consists of an infrared laser projector combined with amonochrome CMOS sensor, which captures video data in 3D under anyambient light conditions. The sensing range of the depth sensor isadjustable, and Kinect software is capable of automatically calibratingthe sensor based on gameplay and the player's physical environment,accommodating for the presence of furniture or other obstacles.

Described by Microsoft personnel as the primary innovation of Kinect,the software technology enables advanced gesture recognition, facialrecognition and voice recognition. According to information supplied toretailers, Kinect is capable of simultaneously tracking up to sixpeople, including two active players for motion analysis with a featureextraction of 20 joints per player. However, PrimeSense has stated thatthe number of people the device can “see” (but not process as players)is only limited by how many will fit in the field-of-view of the camera.

The Virtual Reality Medical Application System of the present, in apreferred embodiment, is particularly suited for use with patientsundergoing radiation therapy, but may be used to cover patientsundergoing brachytherapy, Magnetic Resonance Imaging (MRI), angiography,biopsy procedures and endoscopy such as bronchoscopy or colonoscopy, forexample. In addition to keeping the technician informed of the status ofthe patient, the Virtual Reality Medical Application System of thepresent invention also helps the patient relax and also distract fromthe anxiety and pain of these procedures. The visual and audio systemallows the healhcare provider to share the clinical information to thepatient (e.g., the images of abnormal findings from the colonoscopy)

The Virtual Reality Medical Application System of the present inventionalso provides benefits in for patient setup before each radiationtreatment. When a patient comes in every day for the radiationtreatment, he/she must be positioned in exactly the same position as thereference or planned position. This is performed by using restraineddevices, adjustment by operators/technicians, and then verified by usinginvasive X-ray images or Computed Tomography (CT scan). All of these addtimes (5-10 minutes), cost (hundred of dollars), and unnecessaryradiation exposure to patients. The Virtual Reality Medical ApplicationSystem of the present invention uses body sensor to detect the 3-D bodyposition, comparing to the reference or desired position, and givefeedback to patients via a game avatar to instruct patient to makeadjustment on a specific body parts on their own to get within thereference position. Therefore, it provides a much less expensive,noninvasive (no radiation imaging required), more accurate, and quicker.

The Virtual Reality Medical Application System relies on motion sensors,google, and 3-D game to monitor patients during the radiation treatmentand provide instant feedback to patients to remind them staying stillvia such process as pausing of the game, commands from game character,etc. Since radiation treatment is typically delivered in 15-20 minduring which patients must remain still, any unnecessary involuntarilymovement will cause radiation to deliver incorrectly to normal structureand missing tumor. Within radiation therapy, the Virtual Reality MedicalApplication System of the present invention is particularly well suitedfor use with children with cancer that undergo radiation therapy. Thereare about 10,000 cases of children cancer in the U.S. every year, andabout ½ of these children will undergo radiation therapy daily forseveral weeks.

Because, kids typically can be very anxious, staying still is asignificant problem during these procedures. The alternative to keepingstill voluntarily is to sedate them via intravenous sedatives andanesthetic gas. Sedation comes with anesthesia medication risk, costsseveral thousand dollars per day, and extends daily treatment time fromless than an hour to more than 4 to 5 hours per day. The Virtual RealityMedical Application System of the present will be useful because it willreduce the complications that can come from sedating a child, reducecosts since sedation will not be necessary, and speed up the proceduresince there is no need to wait for the anesthesia and its recovery.

The Virtual Reality Medical Application System of the present inventionis novel and non-obvious because it utilizes a three-dimensional virtualreality game/simulation and sensors to feed the information directly tothe patients to help the patient remain still voluntarily. In apreferred embodiment, the patient will play the game wearing a headmounted display or HMD or Head Mounted Display. The HMD will provide aview into the virtual world where the patient can see himself as anavatar. The HMD will also provide motion tracking of the head and asensor such as a camera will track the movement of the patient's body.When the patient moves in the real world, his avatar will move in thevirtual world. Since, the movement is detected by the game, and thepatient will immediately see that he is moving in real time, the gamewill negatively reinforce the patient to not move utilizing game playmechanics. For example, playing a virtual game of hide and seek, wheremovement and being detected will make you lose the game. This is oneexample of negative reinforcement. There is also positive reinforcementin that has the player completes various levels by remaining still, heis positively reinforce with an achievement. As he builds moreachievements and levels up, he will want to remain still. The virtualreality provided by the goggles and headphone and microphone will helppatients remain relaxed by distracting patients from the stressfulenvironment of the treatment room, hence relieve the anxiety.

The Virtual Reality Medical Application System of the present inventionalso utilizes biofeedback data via sensors such as heart rate, bloodpressure, body motion, EKG, EMG, and EEG to enhance the game experience,and to improve the treatment efficacy to the patient. For instance,using the hide and seek game as an example outlined above, when thecharacter seeking for the patient (e.g. his or her avatar) isapproaching the avatar's hiding place, if the system detects an increaseheart rate or increase blood pressure, the system may have the seekingcharacter retreat from the patient's avatar thus reducing the anxietylevel of the patient and thus increasing the odds of the patient notmoving.

The three-way communication system allows for continuous communicationbetween patient, the healthcare provider, and the simulated game(software).

In an application of the Virtual Reality Medical Application System ofthe present invention associated with a radiation, the radiationtreatment typically takes about 30 minutes overall. However, during this30 minute treatment, there is a crucial window of about 10 minutes wherethe patient should be absolutely still. During this treatment period,the Virtual Reality Medical Application System utilizes innovative gameplay to keep the patient relatively still for 30 minutes but focus onkeeping the patient absolutely still for the crucial time period. In thecase of radiation therapy, this is ten minutes. During this criticalperiod, the game play may have heightened requirements for remainingstill, coupled with increased scoring rates during this critical period,the gamification of the medical process results in a more effectivetreatment.

Virtual Reality Medical Application System of the present invention alsoincludes a detection-notification cycle which is automated because thepatient receives real time feedback to not move. This reduces thecomplications caused by human error. For instance, because the patientis immersed in the virtual environment, he cannot see or hear theexternal environment, including the rather noisy radiation machine orMRI machine. This environmental isolation helps the patient relax andincreases their ability to remain still during the procedure. Moreover,the patients immersion in a 3-D world via a HMD has analgesic propertiesfor acute pain, which provides for a reduction in pain perceived by thepatient.

The Virtual Reality Medical Application System of the present inventionis suitable for a variety of medical applications. For instance, allpatients of all ages that receive radiation therapy (includingbrachytherapy) will have an improved experience using the presentinvention. There are about 700,000-800,000 patients that receiveradiation treatments yearly, and many of these patients are unable toremain still due to pain or anxiety, and which will benefit animprovement in their treatment from the present invention.

The Virtual Reality Medical Application System of the present inventionalso provides benefits when used with respiratory-gating/tracking of atumor during radiation therapy. While a patient is receiving radiationtreatment for the duration of about 10-15 minutes, there is tumor motiondue to respiration, particularly tumors in the lung, breast, and liver.The normal radiation field or treatment portal has to be enlarged toaccount for this tumor motion, which means more normal structure tissuesis being exposed the radiation treatment, hence more treatment sideeffects. One way to reduce the size of treatment portal or radiationfield size is to turn the radiation on only during part of therespiration cycle by having holding their breath for a period of 20-30seconds. This process is called “respiration gating”. With the inputfrom body motion sensor, the present invention facilitates the accuratetiming of the radiation beam (“respiration gating”) with the respiratorycycle to minimize the effect of tumor motion. The body sensor will sendfeedback to patients via the headset goggle to instruct patients to holdtheir breath at the right movement or similar 3-D chest position andsend signal to the treatment machine and/or operator to allow theradiation beam to turn on.

Current systems, such as the Varian® Real-time Position Management™ orRPM system, uses an infrared tracking camera and a reflective marker,the system measures the patient's respiratory pattern and range ofmotion and displays them as a waveform. The gating thresholds are setwhen the tumor is in the desired portion of the respiratory cycle. Thesethresholds determine when the gating system turns the treatment beam onand off. The disadvantages of the RPM system are that it relies only onone or two reflective marker (not the whole 3-D chest motion), patientshas no control or awareness, and expensive, ranging from $100,000 to$150,000. This system also requires hardware installation into thetreatment room.

Another system from Elekta Active Breathhold Coordinator (ABC) systemuses spirometer where patient takes a deep breath and breath slowlythrough their mouth through a tube connected to a spirometer. In thissystem, the Patient does have control and awareness; however, thissystem requires patient cooperation, is subjective and uncomfortablesince the patient has to hold a snorkel-like mouthpiece in theft mouthair tight for 30-40 minutes. It is also expensive, ranging from $70,000to $100,000. The Virtual Reality Medical Application System of thepresent invention also provides benefits to patients using the bodysensors and goggle feedback is non-invasive, more objective, easycompliance, patient-controlled, and less expensive system. This systemalso provides additional sensory input to patients and operators such asoxygen saturation level, heart rate, body temperature, etc.

The Virtual Reality Medical Application System of the present inventionalso helps patients that undergo medical imaging (MRI, CT, PET) that aretoo anxious, such as due to claustrophobia, to remain calm. There areestimated 30 millions of patients receiving MRI yearly, and about 5% ofthese patients have claustrophobia that requires sedation withmedication.

The Virtual Reality Medical Application System of the present inventionis also useful for patients undergoing minor procedures such as biopsy,cardiac angiography, colonoscopy, endoscopy, bronchoscopy, dentalsurgery, cosmetic surgeries, interventional radiology procedures, etc.The Virtual Reality Medical Application System of the present inventionprovides distraction, or escape, from the procedure, thus reducing theneed for sedatives and pain medication. The visual and audio systemallows the healhcare provider to share the clinical information to thepatient (e.g., the images of abnormal findings from the colonoscopy andshare informed consent on what to do with certain findings).

The Virtual Reality Medical Application System of the present inventioncan also allow patients to visualize the real live images of theprocedure that the physicians see on their scope. Since the patients arenot sedated, their physicians can communicate and get patients' consentto certain procedures such as decision to biopsy or removal. This isparticularly beneficial when the lack of patient consent would prohibita physician from performing a procedure that would be routine, but wereunanticipated at the start of the procedure. Moreover, such activepatient participating would eliminate countless repeat medicalprocedures.

Another use of the Virtual Reality Medical Application System of thepresent invention includes psychotherapy for patients that are receivingchemotherapy. These patients typically spend 4-8 hours per day in thechemo-infusion rooms or in the hospital receiving their chemotherapydrugs intravenously. The Virtual Reality Medical Application Systemprovides virtual reality escape or “cancer fighting games” that canrelieve patients' stress, anxiety, and other cancer related symptomssuch as fatigues, nausea, etc.

Also, the Virtual Reality Medical Application System of the presentinvention is suitable for psychotherapy for patients that have acutepains such as those just have surgery, trauma, such as accidents orburns. The Virtual Reality Medical Application System provides a virtualreality escape or “games” that can relieve patients' stress, anxiety,and other related symptoms, as well as provide psychotherapy forpatients that have chronic pain, depression, anxiety disorder, or otherpersonality/mood/affect disorders (autism, OCD, etc. . . . )

Also, the Virtual Reality Medical Application System of the presentinvention is suitable for psychotherapy for patients that suffer painfrom missing arms or legs (“phantom limb pain”). There are approximately70,000 patients in the U.S. who loose their arms/legs due to militarycombat or disease such as diabetic or cancer. These patients sufferchronic pain and depression due to the missing limbs.

Another medical device (VisionRT) uses real-time 3-D surface imagingusing in-room camera to monitor movement. This monitoring is achieved bycomparing the live three dimensional patient surface images withreference images and the system provides the difference between the 2image sets to the technician. When movement is detected and theworkstation will alert the technician, the technician alerts the patientnot to move. This system does not allow patient's participation andmonitoring of their movement, and patients has no control or awareness.If the technician sees body movement, he/she has to re-enter totreatment room to re-adjust patient position to match the referenceposition, and this adds additional time to the treatment. This systemalso requires the monitored body area to be exposed, and it isuncomfortable to some patients (exposing private body areas such asbreast or genitals) or due to the low temperature in the treatment room.This approach is even less effective when patients are experiencing painand uncontrolled movements, as well as those with hearing impairment orare simply too young to understand the technician's instructions. It isalso expensive, ranging from $150,000 to $250,000. This system alsorequires hardware installation into the treatment room.

Similar patient's body position requirement also applies when patientundergoes medical diagnostic imaging such as Computed Tomography (CT) orComputed Axial Tomography (CAT) scan, Magnetic Resonance Imaging (MRI)scan, or Position Emitting Tomography (PET) scan. These diagnosticmedical imaging procedures typically lasts about 30-90 minutes, andrequire patient to lie in a certain position during that period. Theseimaging devices have a small space and create a feeling of “beingtrapped” (claustrophobia), it creates anxiety and uncomfortableenvironment for patient to relax. Millions of people suffer fromclaustrophobia in U.S. alone, and some could not tolerate these imagingprocedures necessary for their medical care or screening. Involuntarybody movement will affect the image quality leading to incorrect,uncertain or inconclusive findings. This then often requires repeatimaging or additional imaging procedure or medical procedure which addscost and inconvenience to patient and health care practitioner. Onecurrent option is to ask patient to close their eyes or using a eyepatch and listening to music to relax. Another option is to use a videogoggle with headphone to show a video or movie to distract the patient.This video goggle does not affect imaging devices function, andtypically cost about 30,000 to 40,000 per system, and required somehardware installation to the room.

Patients undergo minor medical procedures such as biopsy, angiography,or medical endoscopy, where a thin flexible fiber optic scope orcatheter is inserted into their body to allow clinicians to visuallyinspect an area of disease, obtain medical imaging using video,photography, ultrasound or X-ray, obtain a biopsy of body tissue, orprovide a medical treatment such as cauterization, laser treatment, orputting in a stent to keep an area open. Depending on the organ beingexamined, these endoscopy procedures can have different names such asbronchoscopy (for lung), colonoscopy (for colon and rectum), proctoscopy(for rectum), anoscopy (anus), cystoscopy (for bladder and urethra),Esophago-Duodeno-Gastroscopy (esophagus, stomach, and small bowel),hysteroscopy (cervix, uterus), endoscopic retrogradecholangiopancreatography or ERCP (for bile duct and pancreas),laryngoscopy (larynx). With the angiography, where catheter isintroduced into the patient's blood vessel to make diagnosis andtreatment of blood vessel disease such as obstruction. Patientsundergoing these procedures require sedation and pain medication viaintravenous route and local pain medication (local anesthetics).Occasionally, during these procedures, clinicians want to share theendoscopy findings to patients and getting patients' feedback on furtherprocedure, but not able to due to patients being sedated.

In one aspect, systems and methods are disclosed for a patientmonitoring system that monitors both the biometric and physicalcondition of a patient, and provides a virtual reality interface to thatpatient to improve the tolerance and efficacy of the medical treatmentin a cost-effective manner. This system allows patients' participationand monitoring of their movement, and patients has control and awarenessof their body position, and can make adjustment to the referenceposition. Both the patient and treating technician can see and verifythe body positioning. The system is mobile, can be implemented easilyand does not require any hardware installation to the treatment room.

In another aspect, systems and methods are disclosed for a virtualreality environment which interfaces to medical treatment apparatus andwith medical service providers to share the medical findings and todecrease the discomfort, anxiety and pain of a patient during aprocedure, such as radiation therapy, medical imaging (CT, MRI, PET),medical endoscopy procedures (bronchoscopy, colonoscopy, proctoscopy,cystoscopy, etc) or biopsy.

In yet another aspect, systems and methods are disclosed for monitoringa patient by positioning the patient for a predetermined medicalmission; sensing biometric and physical conditions of a patient duringthe mission, and displaying a multimedia interaction with the patient tokeep the patient in a predetermined position to improve efficacy of amedical mission.

In a further aspect, the Virtual Reality Medical Application System is amedical system including a combination of hardware and software thatwill improve a patient's ability to remain still during medicalprocedures that require a patient to not move. The system includes threecomponents: the hardware component; the software component; and a sensorcomponent. The hardware component consists of a three dimensional (3-D)goggle to provide visual images to patients, noise-cancellationheadphone and microphone system to provide 3-way communication betweenthe patient, the healthcare provider, and the Virtual Reality MedicalApplication System software.

Implementations of the above aspects can include one or more of thefollowing. The medical mission can be an operation, a treatment, abiological sampling, an irradiation process, medical endoscopy,angiography, or a body medical imaging scan. The system can render avisualization with a virtual reality display to interact with thepatient or the real image of the findings from the endoscopy. Visualimages can be provided to patients with a three dimensional (3-D)goggle. The system can render sound to patients with anoise-cancellation headphone and microphone system to provide a 3-waycommunication between the patient, a healthcare provider, and a VirtualReality Medical Application System software.

The system can render a patient specific 3-D game controlled by thepatients. The patient-controlled game environment can provide a gooddistraction to relieve pain and anxiety. A virtual simulation of reallife scenery can be provided or the real image or video of the procedurecan be shown to the patient. The system can render images of a procedurethat a healthcare provider wants to share with the patient.

The sensing can include tracking motion or capturing biofeedback data.The motion tracking uses gyroscope and accelerometer sensors on a 3-Dgoggle to detect head motion and an array of Inertial Measurement Units(IMU) are used to track the body. IMU's are electrical componentsutilizing a combination of gyro meters, accelerometers, andmagnetometers to track motion in 3 dimensional space. IMUs are placed onvarious parts of the body to monitor patient body motion. The system candetect one or more of blood pressure, heart rate, EEG, and EKG. Thesystem can include treatment for one of: radiation therapy,brachytherapy, Computed Tomotherapy (CT), Positron Emission Tomotherapy(PET), Magnetic Resonance Imaging (MRI), angiography, biopsy andendoscopy. The virtual reality display relaxes or distracts the patientfrom anxiety or pain. The system can share clinical information with thepatient using the multimedia interaction. The system can guide a patientinto a reference position for setup before each treatment. The systemcan detect a patient position using a 3-D body sensor, comparing thepatient position to the reference position, and providing feedback tothe patient to move to the reference position. Patients can play a gamethat negatively reinforces the patient to not move utilizing game playmechanics. Alternatively, the system can provide levels of positivereinforcement to reward the patient for remaining still.

In yet other embodiments, the Virtual Reality Medical Application Systemof one embodiment utilizes a three-dimensional virtual realitygame/simulation and sensors to feed the information directly to thepatients to help the patient remain still voluntarily. In a preferredembodiment, the patient will play the game wearing a head mounteddisplay or HMD or Head Mounted Display. The HMD will provide a view intothe virtual world where the patient can see himself as an avatar. TheHMD will also provide motion tracking of the head and a sensor such as acamera will track the movement of the patient's body. When the patientmoves in the real world, his avatar will move in the virtual world.Since, the movement is detected by the game, and the patient willimmediately see that he is moving in real time, the game will negativelyreinforce the patient to not move utilizing game play mechanics. Forexample, playing a virtual game of hide and seek, where movement andbeing detected will make you lose the game. This is one example ofnegative reinforcement. There is also positive reinforcement in that hasthe player completes various levels by remaining still, he is positivelyreinforce with an achievement. As he builds more achievements and levelsup, he will want to remain still. The virtual reality provided by thegoggles and headphone and microphone will help patients remain relaxedby distracting patients from the stressful environment of the treatmentroom, hence relieve the anxiety and discomfort or pain.

The Virtual Reality Medical Application System of one embodiment of theinvention also utilizes biofeedback data via sensors such as heart rate,blood pressure, body motion, EKG, EMG, and EEG to enhance the gameexperience, and to improve the treatment efficacy to the patient. Forinstance, using the hide and seek game as an example outlined above,when the character seeking for the patient (e.g. his or her avatar) isapproaching the avatar's hiding place, if the system detects an increaseheart rate or increase blood pressure, the system may have the seekingcharacter retreat from the patient's avatar thus reducing the anxietylevel of the patient and thus increasing the odds of the patient notmoving.

The three-way communication system allows for continuous communicationbetween patient, the healthcare provider, and the simulated game(software).

In an application of the Virtual Reality Medical Application System ofone embodiment of the invention associated with a radiation, theradiation treatment typically takes about 20-30 minutes overall.However, during this 20-30 minute treatment, there is a crucial windowof about 10 minutes where the patient should be absolutely still. Duringthis treatment period, the Virtual Reality Medical Application Systemutilizes innovative game play to keep the patient relatively still for30 minutes but focus on keeping the patient absolutely still for thecrucial time period. In the case of radiation therapy, this is tenminutes. During this critical period, the game play may have heightenedrequirements for remaining still, coupled with increased scoring ratesduring this critical period, the gamification of the medical processresults in a more effective treatment.

Virtual Reality Medical Application System of one embodiment includes adetection-notification cycle which is automated because the patientreceives real time feedback to not move. This reduces the complicationscaused by human error. For instance, because the patient is immersed inthe virtual environment, he or she cannot see or hear the externalenvironment, including the rather noisy radiation machine or MRI machineor other medical equipment and environment. This environmental isolationhelps the patient relax and increases their ability to remain stillduring the procedure. Moreover, the patient's immersion in a 3-D worldvia a HMD has analgesic properties for acute pain, which provides for areduction in pain perceived by the patient.

Advantages of the above aspects and implementations may include one ormore of the following. The apparatus and method decrease the movement ofpatients undergoing medical procedures such as radiotherapy. The systemadvantageously provides an apparatus and method to decrease the movementof patients undergoing medical procedures such as radiotherapy whichdoes not use physical restraints or chemical sedatives to immobilize apatient. The apparatus and method decrease the movement of patientsundergoing medical procedures such as radiotherapy which is non-invasiveand comfortable. The apparatus and method decrease the movement ofpatients undergoing medical procedures such as radiotherapy whichmonitors the real-time position of a patient and relays the informationto the patient about the specific area of the body and allows patient tomake adjustment to get back to the reference or desired position. Theapparatus and method decrease the movement of patients undergoingmedical procedures such as radiotherapy which alerts a third party(operator or technician) of excessive movement outside a set range andwhich part of the body. The apparatus and method assure the safely topatients undergoing medical procedures such as radiotherapy which sendan interrupt signal instantly to the radiation equipment to pause theradiation beam due to excessive movement outside a set range. Theapparatus and method decrease the movement of patients undergoingmedical procedures such as radiotherapy which provides a virtual realitysystem which visualizes the patient's current physical and biometricmeasurements and allows for the real-time adjustment of their physicaland biometric measurements in response.

Yet other advantages may include one or more of the following. Thesystem enables tight control of radiation exposure for children aschildren are naturally more anxious than adults. Staying still forprolonged periods of times for the treatment is a difficult propositionfor children and the use of mechanical stabilization devices orsedatives is commonly used. The mechanical stabilization devicesgenerally cause additional anxiety in children as they are bound andimmobilized, resulting in crying, screaming, and minute movements asthey struggle to become free. The Virtual Reality Medical ApplicationSystem provides a virtual reality escape or “games” coupled with EEG andEMG sensors that can relieve patients' chronic pain and depression.

The use of sedatives, although effective to immobilize, comes withmedical risk of the sedative medications, costs (up to several thousanddollars per treatment), and extended induction and recovery time (3-5hours per treatment). This risk is repeated every day, days per week,for period of 3-6 weeks. The system's game and role playing modes allowchildren patients to be comfortable during radiation treatment sessionsin a way that is cost-effective, drug free, restraint free, and funmanner.

Overall, the Virtual Reality Medical Application System of the presentinvention provides for high levels of patient safety, provides for apatient controlled treatment session, provides non-invasive patientposition monitoring using no X-ray scanning, automated patient positionmanagement without requiring any operator involvement, and provides forinstant communication with a patient during treatment, and provides anoverall cost-savings to traditional treatment approaches.

BRIEF DESCRIPTION OF THE DRAWING

The nature, objects, and advantages of the present invention will becomemore apparent to those skilled in the art after considering thefollowing detailed description in connection with the accompanyingdrawings, in which like reference numerals designate like partsthroughout, and wherein:

FIG. 1 is a system level diagram of the Virtual Reality MedicalApplication System of one embodiment of the present invention includinga Virtual Reality Console having a video receiver and processor formonitoring patient movement, a biometric receiver and processor formonitoring patient biometric conditions, and patient information anduser interface including a virtual reality headset and patientcontroller, in combination with an array of motion sensors, biometricdata collection devices, and a medical treatment apparatus, such as anX-ray radiation treatment device or imaging devices (CT, MRI, PET),having a treatment selection database, patient biometric database, andpatient physical database;

FIG. 2 is a top plan view of an exemplary patient positioned flat on herback on a treatment table of FIG. 1, and having a number of biometricsensing devices positioned to sense a variety of biometric signallevels, a virtual reality headset with headphone and microphone, one ormore patient controllers to receive patient input, a number of imagingcameras positioned to detect patient movement, and a number of positionlocator tags distributed on the patient's body for precise positiondetection, and a representative radiation target on her abdomenindicating the location of the radiation treatment or medical procedureto be performed;

FIG. 3 is a system level block diagram of the hardware of the VirtualReality Medical Application System of one embodiment of the presentinvention showing the console sub-system in communication with thevirtual reality head mounted display (HMD), the patient input controllersubsystem having a variety of user input devices to capture patientinput and responses, the patient motion detection subsystem having avariety of motion sensor inputs to detect patient motion, thebiofeedback sensor subsystem having a variety of biometric sensor inputsto maintain measurement of critical patient biometric data, and thetechnician's workstation to provide real-time data to a treatmenttechnician;

FIG. 4 is a system level block diagram of the software of the VirtualReality Medical Application System of one embodiment of the presentinvention showing a module subsystem for the user interface andcharacter behavior to be used in game animations and patient feedback,an engine subsystem that provides patient feedback animations,renderings, and motion simulations, and an associated HMD interface thatconnects to a game module subsystem and provides graphics and patienthead motion tracking and live video to the patient related to the videoprocedure, a communication interface which interconnects the variousmodules, a motion detection interface receiving motion data frommultiple motion detection imagers and devices, and a biofeedbackinterface receiving biometric data from the various patient sensors, allin communication with a technician console interface and relatedtechnician console;

FIG. 5 is an exemplary flow chart depicting the pre-treatment simulationof the treatment using one embodiment of the Virtual Reality MedicalApplication System of the present invention, including acquiring patientidentification data, establishing baseline biometric and position datato be used in subsequent treatments;

FIG. 6 is a flow chart representing an exemplary operation of theVirtual Reality Medical Application System of one embodiment of thepresent invention including the preliminary steps executed prior tobeginning treatment, including initializing the virtual reality system,customizing the system for the particular patient with current biometricand physical data, establishing data limits, and initiating thepatient's virtual reality treatment protocol;

FIG. 7 is a flow chart representing the steps for the acquisition ofbaseline patient biometric data of FIG. 5, and including the accessingof the patient's historical data, and measuring and storing all patientbiometric data, such as 3-D spatial positioning, skin conductivity, EKG,EEG, heart rate, respiration rate, body temperature and oxygensaturation, for example;

FIG. 8 is a flow chart representing the steps for the acquisition ofbaseline physical positioning data of FIG. 6, and including theaccessing of the patient's historical data and measuring and storing allpatient physical data, and then comparing a patient's current positionto his baseline data, and providing feedback to the patient to adjusthis position to match the baseline data;

FIG. 9 is a flow chart representing the beginning of treatment of FIG. 6without using patient controller input, and includes the verification ofpatient identity, the activation of the medical treatment apparatus, andthe subsequent monitoring and evaluation of position and biometric datafor compliance and safety, and providing a patient with biometric andpositional feedback to assist with correction, and modification of thevirtual reality program to facilitate such patient correction;

FIG. 10 is a flow chart representing the beginning of treatment of FIG.6 using patient controller input, and includes the verification ofpatient identity, the activation of the medical treatment apparatus, andthe subsequent monitoring and evaluation of position and biometric datafor compliance and safety, providing a patient with biometric andpositional feedback to assist with correction, receipt of patient inputdata, and modification of the virtual reality program to facilitate suchpatient correction;

FIG. 11 is an exemplary flow chart depicting operation of the gameinterface of the Virtual Reality Medical Application System of thepresent invention, and includes a language selection, display of a gametitle and avatar selection, providing operating instructions to thepatient confirming the patient's understanding, providing a tutorial,and then providing a patient interactive game sequence intended toassist a patient to minimize movement during the procedure, andproviding patient incentives for being still including game tokenrewards and scoring;

FIG. 12 is an exemplary view of the patient's display image in the headmounted display during the operation of the Virtual Reality MedicalApplication System of one embodiment of the present invention anddepicting various video progressions of a child avatar playing in a parkand being sought out by a friendly puppy, scoring points for successfulavoidance of the exuberant puppy, and a high scoring segmentcorresponding to a critical treatment period requiring the patient toremain still as the puppy continues to seek the avatar;

FIGS. 13 through 22 depict an alternative sequence of a game displayedin the Virtual Reality Medical Application System of the presentinvention, which is an animated story in which a patient-linked avatarhelps a dinosaur to retrieve its eggs from an egg-stealing robot byriding a flying skateboard throughout a virtual world to retrieve thestolen eggs, and requires the avatar to remain motionless during flightto avoid losing the trail of the stolen eggs, or dropping any retrievedeggs, thus returning all of the stolen eggs safely to the gratefuldinosaur;

FIG. 13 includes an exemplary title page in FIG. 13 including a gamename, and representative game graphics to introduce the patient to thegame environment;

FIG. 14 is an exemplar of various game avatar characters, such as aneccentric doctor, a dinosaur, and a boy and girl (patient) avatar thatcan be used in the game of the Virtual Reality Medical ApplicationSystem of the present invention;

FIG. 15 is an exemplary avatar selection screen in which an eccentricdoctor selects a boy or girl (patient) avatar, such as by the patientrotating his or her head in the direction of the desired avatar, such asturning the head right to select the girl avatar, or left to select theboy avatar, with the selection being advanced to the next game screen;

FIG. 16 is an exemplary display of a game within the Virtual RealityMedical Application System of one embodiment of the present inventiondepicting the patient avatar on a flying skateboard and travelingthrough a virtual forest in pursuit of lost dinosaur eggs;

FIG. 17 is an exemplary map of a game within the Virtual Reality MedicalApplication System of one embodiment of the present invention showing arepresentative forest with colorful topography, and the advancement ofthe patient avatar from an origin to an end point designed to allow theavatar to gather the stolen eggs from the forest;

FIG. 18 is an exemplary display of the game showing the egg-stealingrobot, the dinosaurs, and the patient avatar hiding from the robot inorder to avoid detection to protect the dinosaur eggs;

FIG. 19 is an exemplary display of the game showing the patient avatarriding a flying skateboard through the virtual forest environment insearch for stolen dinosaur eggs, and providing a patient with a motionstatus indicator light on the skateboard corresponding to patientmovement measurement feedback to the patient, such as green for good,yellow for minimal patient motion detected, and red for too much motiondetected, and an exemplary map identifies where the patient avatar iswithin the virtual forest path;

FIG. 20 is a representative game end screen reporting the patient'ssuccess in collecting all of the stolen eggs from the virtual forest;

FIG. 21 is a representative game end screen reporting the patients'failure in collecting all of the stolen eggs from the virtual forest;

FIG. 22 is representative display of the array of IMU for body motiondetection;

FIG. 23 is a representative display of the IMU for the respiratorygating application; and

FIG. 24 is an exemplary display of the game showing how the game canassist the patient in holding their breath and breathing properly forrespiratory gating application.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT

Hardware System Description

Referring initially to FIG. 1, a system level diagram of the VirtualReality Medical Application System of the present invention is shown andgenerally designated 100. System 100 is intended to treat a patient 102,and includes a treatment apparatus 104, such as a radiation-deliveringX-ray system or medical imaging MRI system. A Virtual Reality Console106 interfaces to a user, or technician, controller 108 to cooperatewith the treatment apparatus 104.

System 100 also includes a number of imaging devices 110, 112, 114, and116, each having a field of view 117 (shown in dashed lines) to perceivemovement of patient 102. Signals from imaging devices 110, 112, 114, and116 provided data along connection 118 to the video image receiver 120in VR Medical Console 106, which also includes a video image processor122 to process the video images to detect patient movement and monitorpatient position. A biometric receiver 124 is provided and includes aprocessor for monitoring patient biometric conditions including themotion sensor for 3-D spatial position. Console 106 also includes avirtual reality video driver 126 and virtual reality audio driver 128.

A treatment protocol database 130, biometric data database 132 andphysical data database 134 stores and provide data for patienttreatment, and can provide specific patient data from prior treatmentsessions, as well as store current treatment data for use later. Also,all treatment data may be stored in hard drive 140.

A patient controller receiver 136 and game software 138 reside in VRmedical console 106 to provide the virtual reality experience forpatient 102. Console 106 interfaces with the virtual reality headset,headphone and microphone via input 142, and biometric data is providedto console 106 via input 146, and patient controller input is receivedon input 148.

A VR Medical Application controller 108 includes a treatment selectiondatabase 152, a patient biometric database 154 and a patient physicaldatabase 156, and may include a computer console 150 for the technicianto operate and interface with the Reality Medical Application System ofthe present invention. Control signals are provided from console 150 totreatment apparatus 104 via channel 170, and may include safetyfeatures, such as device interrupts, and error condition notifications.

Data may be exchanged between console 106 and user controller 108. Forinstance, control inputs 160 including historical data and patient data,alerts 162 including interrupts, biometric data and physical data, andbidirectional channel 164 including audio and video signals. These datachannels provide a technician seated apart from the patient the abilityto fully monitor the patient 102, and the patient's interaction with theVirtual Reality Medical Application System and the treatment apparatus104.

Radiation from treatment apparatus 104 causes cellular degradation dueto damage to DNA and other key molecular structures within the cells invarious tissues. It was discovered that with controlled amounts ofradiation, the effects caused by radiation can be utilized to treatcertain diseases such as cancer. With the use of radiation, severalpreviously untreatable diseases have been able to be successfullytreated using radiation therapies. However, along with the curativeeffects of radiation are negative side effects. The prolongedunnecessary exposure to radiation to normal organs includes bothimmediate and delayed side effects. The immediate side effects ofradiation exposure include nausea and vomiting, diarrhea, pain,headaches, and skin or organ burns, whereas the delayed effects couldinclude fatigue, weakness, hemorrhaging, leukopenia, epilation,paralysis, brain damage, blindness, perforation, ulceration,malabsorption, organ failure, and other various side effects.Additionally, the prolonged overexposure of unnecessary radiation maycause more serious complications and may lead to mortality. The systemof FIG. 1 reduces the serious side effects of utilizing radiation as aform of medical treatment, and limit the exposure of radiation tohealthy tissue by focusing the radiation solely on the diseased tissueby training patients to remain still so that the radiation can befocused on diseased portions, and minimizing the radiation exposure tothe uninvolved healthy organs.

The Virtual Reality Medical Application System of one embodiment of theinvention is suitable for a variety of medical applications. Forinstance, all patients of all ages that receive radiation therapy(including brachytherapy) will have an improved experience using thepresent invention. There are about 700,000-800,000 patients that receiveradiation treatments yearly, and many of these patients are unable toremain still due to pain or anxiety, and which will benefit animprovement in their treatment from the present invention.

The Virtual Reality Medical Application System of one embodiment alsoprovides benefits when used with respiratory-gating/tracking of a tumorduring radiation therapy. While a patient is receiving radiationtreatment for the duration of about 10-15 minutes, there is tumor motiondue to respiration, particularly tumors in the lung, breast, and liver.The normal radiation field or treatment portal has to be enlarged toaccount for this tumor motion, which means more normal structure tissuesis being exposed the radiation treatment, hence more treatment sideeffects. One way to reduce the size of treatment portal or radiationfield size is to turn the radiation on only during part of therespiration cycle by having holding their breath for a period of 20-60seconds. This process is called “respiration gating”. With the inputfrom body motion sensor, the present invention facilitates the accuratetiming of the radiation beam (“respiration gating”) with the respiratorycycle to minimize the effect of tumor motion. The body sensor will sendfeedback to patients via the headset goggle to instruct patients to holdtheir breath at the right movement or similar 3-D chest position andsend signal to the treatment machine and/or operator to allow theradiation beam to turn on.

The Virtual Reality Medical Application System of one embodiment alsoprovides benefits to patients using the body sensors and goggle feedbackis non-invasive, more objective, easy compliance, patient-controlled,and less expensive system. This system also provides additional sensoryinput to patients and operators such as 3-D body spatial position,oxygen saturation level, heart rate, body temperature, etc.

The software component provides a variety of patient specific 3-D gamesthat are controlled by patients, virtual simulation of real life scenery(car racing, hot-air balloon, fish-eye view, etc), and may include realtime images of the procedure that a healthcare provider wants to sharewith patient, such as an inside image of the colon from the colonoscopy.The proposed system can be used to decrease the discomfort, anxiety andpain patients during procedures, such as, medical imaging (CT, MRI,PET), medical endoscopy procedures (bronchoscopy, colonoscopy,proctoscopy, cystoscopy, etc) or biopsy.

The 3-D games and virtual simulation software will be developed is, in apreferred embodiment, run on Linux, and the technician console is across platform and runs on Microsoft Windows, Apple Mac OS, and Linux ina custom medical-grade computer with input/output ports and storagesystem.

The sensor component performs two major functions: motion tracking andbiofeedback. For motion tracking of the patients, the head and body aretracked separately. The tracking of the head and the body both utilizesensors based on Inertial Measurement Units (IMU). IMU's are electricalcomponents utilizing a combination of gyro meters, accelerometers, andmagnetometers to track motion in 3 dimensional space. For the head, theIMU's are built into the 3-D goggles. For the body, 3D motion is trackedvia an array of IMUs that are placed on the various joints of the body.The array of IMUs transmit data to a central hub and then sent to theVirtual Reality Medical Application System. For biofeedback, sensorssuch as blood pressure, heart rate, EEG, and EKG, among others will helpa technician keep informed of the patient condition, and will tend todetermine the game play mechanics.

Alternatively, motion may be detected using a video based motion sensingdevice, capable of three-dimensional motion detection. One such sensoris commercially available as the Kinect sensor. Kinect builds onsoftware technology developed internally by Rare, a subsidiary ofMicrosoft Game Studios owned by Microsoft, and on range cameratechnology by Israeli developer PrimeSense, which developed a systemthat can interpret specific gestures, making completely hands-freecontrol of electronic devices possible by using an infrared projectorand camera and a special microchip to track the movement of objects andindividuals in three dimensions. This 3D scanner system, often calledLight Coding, employs a variant of image-based 3D reconstruction.

The Kinect sensor is a horizontal bar connected to a small base with amotorized pivot and is designed to be positioned lengthwise above orbelow the video display. The device features an “RGB camera, depthsensor and multi-array microphone running proprietary software” whichprovide full-body 3D motion capture, facial recognition and voicerecognition capabilities. Kinect sensor's microphone array enables itsattached devices, such as an Xbox 360, to conduct acoustic sourcelocalization and ambient noise suppression, allowing for things such asheadset-free party chat over Xbox Live.

The depth sensor consists of an infrared laser projector combined with amonochrome CMOS sensor, which captures video data in 3D under anyambient light conditions. The sensing range of the depth sensor isadjustable, and Kinect software is capable of automatically calibratingthe sensor based on gameplay and the player's physical environment,accommodating for the presence of furniture or other obstacles.

The Virtual Reality Medical Application System of the present, in apreferred embodiment, is particularly suited for use with patientsundergoing radiation therapy, but may be used to cover patientsundergoing brachytherapy, Computed Tomography (CT), PET, MagneticResonance Imaging (MRI), angiography, biopsy procedures and endoscopysuch as bronchoscopy or colonoscopy, for example. In addition to keepingthe technician informed of the status of the patient, the VirtualReality Medical Application System of one embodiment of the inventionalso helps the patient relax and also distract from the anxiety and painof these procedures. The visual and audio system allows the healthcareprovider to share the clinical information to the patient (e.g., theimages of abnormal findings from the colonoscopy) The Virtual RealityMedical Application System of one embodiment of the invention alsoprovides benefits in for patient setup before each radiation treatment.When patient comes in every day for the radiation treatment, he/she mustbe positioned in exactly the same position as the reference or plannedposition. This is performed by using restrained devices, adjustment byoperators/technicians, and then verified by using invasive X-ray imagesor Computed Tomography (CT scan). All of these add times (5-10 minutes),cost (hundred of dollars), and unnecessary radiation exposure topatients. The Virtual Reality Medical Application System of oneembodiment of the invention uses body sensor to detect the 3-D bodyposition, comparing to the reference or desired position, and givefeedback to patients via a game avatar to instruct patient to makeadjustment on a specific body parts on their own to get within thereference position. Therefore, it provides a much less expensive,noninvasive (no radiation imaging required), more accurate, andtime-efficient. The benefit of IMU based motion tracking over videobased motion (kinect) tracking is IMU based motion tracking does notrequire a patient's skin to be exposed so that treatment can be donewhile the patient is covered. With video based motion capture, the bodyneeds to be exposed, which could be uncomfortable for patients withcertain area such as breast and pelvic regions.

The Virtual Reality Medical Application System relies on motion sensors,goggle, and 3-D game to monitor patients during the radiation treatmentand provide instant feedback to patients to remind them staying stillvia such process as pausing of the game, commands from game character,etc. Since radiation treatment is typically delivered in 15-30 minduring which patients must remain still, any unnecessary involuntarilymovement will cause radiation to deliver incorrectly to normalstructures and missing tumor. Within radiation therapy, the VirtualReality Medical Application System of one embodiment of the invention isparticularly well suited for use with children with cancer that undergoradiation therapy. There are about 10,000 cases of children cancer inthe U.S. every year, and about ½ of these children will undergoradiation therapy daily for several weeks.

Because, kids typically can be very anxious, staying still is asignificant problem during these procedures. The alternative to keepingstill voluntarily is to sedate them via intravenous sedatives andanesthetic gas. Sedation comes with anesthesia medication risk, costsseveral thousand dollars per day, and extends daily treatment time fromless than an hour to more than 4 hours per day. The Virtual RealityMedical Application System of the present will be useful because it willreduce the complications that can come from sedating a child, reducecosts since sedation will not be necessary, and speed up the proceduresince there is no need to wait for the anesthesia and its recovery.

The Virtual Reality Medical Application System of one embodiment alsohelps patients that undergo medical imaging (MRI, CT, and PET) that aretoo anxious, such as due to claustrophobia, to remain calm. There areestimated 30 million of patients receiving MRI yearly, and about 5% ofthese patients have claustrophobia that requires sedation withmedication. The Virtual Reality Medical Application System of oneembodiment is also useful for patients undergoing minor procedures suchas biopsy, angiography, colonoscopy, endoscopy, bronchoscopy, dentalsurgery, cosmetic surgeries, interventional radiology procedures, etc.The Virtual Reality Medical Application System of one embodiment of theinvention provides distraction, or escape, from the procedure, thusreducing the need for sedatives and pain medication. The visual andaudio system allows the healthcare provider to share the clinicalinformation to the patient (e.g., the images of abnormal findings fromthe colonoscopy and share informed consent on what to do with certainfindings). The Virtual Reality Medical Application System of oneembodiment can also allow patients to visualize the real live images ofthe procedure that the physicians see on their scope. Since the patientsare not sedated, their physicians can communicate and get patients'consent to certain procedures such as decision to biopsy or removal.This is particularly beneficial when the lack of patient consent wouldprohibit a physician from performing a procedure that would be routine,but were unanticipated at the start of the procedure. Moreover, suchactive patient participating would eliminate countless repeat medicalprocedures.

Another use of the Virtual Reality Medical Application System of oneembodiment of the invention includes psychotherapy for patients that arereceiving chemotherapy. These patients typically spend 4-8 hours per dayin the chemo-infusion rooms or in the hospital receiving theirchemotherapy drugs intravenously. The effect of chemotherapy drugs tothe brain and exposing to the environment can affect patients' cognitivefunctions (memory, fluency, attention, motor coordination) and causedepression, anxiety, hopelessness. This condition is sometime called“chemo-brain”. The Virtual Reality Medical Application System providesvirtual reality escape or “cancer fighting games” that can relievepatients' stress, anxiety, and other cancer related symptoms fromchemo-brain effects.

Also, the Virtual Reality Medical Application System of one embodimentis suitable for psychotherapy for patients that have acute pains such asthose just have surgery, trauma, such as accidents or burns. The VirtualReality Medical Application System provides a virtual reality escape or“games” that can relieve patients' stress, anxiety, and other relatedsymptoms, as well as provide psychotherapy for patients that havechronic pain, depression, anxiety disorder, or otherpersonality/mood/affect disorders (autism, OCD, etc. . . . ) Also, theVirtual Reality Medical Application System of one embodiment of theinvention is suitable for psychotherapy for patients that suffer painfrom missing arms or legs (“phantom limb pain”). There are approximately70,000 patients in the U.S. who lose their arms/legs due to militarycombat or disease such as diabetic or cancer. These patients sufferchronic pain and depression due to the missing limbs. The VirtualReality Medical Application System provides a virtual reality escape or“games” coupled with body motion sensors, EEG and EMG sensors that canrelieve patients' chronic pain and depression.

Overall, the Virtual Reality Medical Application System of oneembodiment provides for high levels of patient safety, provides for apatient controlled treatment session, provides non-invasive patientposition monitoring using no X-ray scanning, automated patient positionmanagement without requiring any operator involvement, and provides forinstant communication with a patient during treatment, and provides anoverall cost-savings to traditional treatment approaches. Moreover, this“patient control” experience, as compared to other systems where patientis “passive”, provides an overall improvement to the patient experience,and allows for the real-time sharing of information and communicationwith the patient. This provides a more efficient procedure, particularlywhen a procedure requires the need to obtain patient consent duringtreatment, which is problematic and time-consuming when the patient isunder anesthesia.

Referring to FIG. 2, a top plan view shows an exemplary patient 102positioned flat on her back on a treatment table 240 of FIG. 1. Motiondetectors 110 having a field of view 111, and motion detector 116 havinga field of view 117 monitor the patient's position. A virtual realityhead mounted display (HMD) 202 is positioned over the patient's eyes,and provides a visual interface for the patient to see duringtreatments. A Virtual reality headset interface having a headphone andmicrophone 204 receives and transmits signals from the HMD 202 to VRmedical console 106 along interface 142 (shown in FIG. 1).

FIG. 2 also shows a number of biometric sensing devices positioned tosense a variety of biometric signal levels. For instance, an EEG sensoroutput 210, EMG sensor output 212, blood pressure sensor 214 providesdata to blood pressure sensor output 216, EKG sensor 218 provides an EKGsensor output 220, skin temperature sensor output 222, and oxygen sensoroutput 224 are channeled to VR medical console 106 as shown in FIG. 1.

Also, a number of motion sensors 232 may be positioned on patient 102 toprovide mechanical measurement of the patient during treatment, and mayinclude a number of different measurement techniques includinggyroscopes, strain gauges, Hall-effect sensors, and other techniquesknown in the art. The signals from the mechanical measurement devicesare provided to motion sensor output 234 for routing to VR medicalconsole 106.

In addition to video imaging devices 116 and 110, and mechanical motionsensing devices 232, a number of position locator tags, or referencemarkers, 226, 228 and 230 may be positioned on the patient. Thesemarkers may have wavelength-specific reflection properties (such asinfrared or ultraviolet), enabling an imaging system to very specificfocus on the patient position using the markers, and comparing thosemarker positions to known references positions.

In one embodiment of the present invention, motion sensors 116 may be ofthe type utilized in the Kinect2 system. Incorporation of such as systemprovides several benefits: (a) It is infrared laser projector and a 3-Dcamera to capture the 3-D body surface; (b) It enables facialrecognition; and (c) It enables voice recognition. The (b) and (c)features are very important for patient safety to correctly identifyingthe correct patient being treated. This is particularly advantageous incritical health care treatment where 130,000 medical mistakes occurannually in which patients receive the wrong surgery or radiationtreatment.

A representative radiation target may be placed on the patient abdomento indicate the location of the radiation treatment, or this target maybe optically generated by the treatment device, such as a radiationemitter.

Referring now to FIG. 3, a system level block diagram of the hardware ofthe Virtual Reality Medical Application System of the present inventionis shown and generally designated 300. System 300 includes a controlsubsystem 302 which runs the game engine, sends data to the technicianworkstation, and incorporates a game console on a personal computer ortablet computer.

Console sub-system 302 is in communication via channel 306 with thevirtual reality head mounted display (HMD) 304 which shields thepatients eyes from the external environment, tracks movement of thepatient's head and is used during game play, and provides the threedimensional stereoscopic display for the patient's virtual environment.In a preferred embodiment of the HMD 304, noise cancelling headphonesmay be incorporated to provide a sound-proof environment for thepatient, and may include a microphone for bi-directional communicationwith the technician at the console 108.

A patient input controller 310 captures patient input to allow thepatient to navigate in the virtual world, and allows the patient toutilize and navigate through the user interface. In a preferredembodiment, the patient input controller can include one or more of eyemovement sensors, a mouse, joystick, gamepad, touchpad, button, voiceinput, and gesture input; with such input signals received on input 312.

A motion detection subsystem 320 detects the patient motion from motioninput devices, such as a video input, kinetic sensors, strain gauges,IMs motion sensors and accelerometers on signal line 322, and providesthe motion-related data to the game software 138 running in the console106, and can also provide data to the technician workstation 108. Thesemotion sensing devices provide highly accurate patient position andmotion data to motion detection subsystem 320 which in turn communicatesthis data to console subsystem 302 using interface 324.

Biofeedback sensor subsystem 326 is in communication with a variety ofbiometric monitoring devices, such as blood pressure monitors, EMG, EKG,EEG, heart rate, respiration rate, body temperature, skin conductivity,and oxygen level monitors along input channel 328. These datameasurements may be returned to the console subsystem 302 via channel330.

The constant measurement of critical patient biometric data provides theVirtual Reality Medical Application System of the present invention toaccurately monitor the physical and mental condition of the patientbefore, during, and after a treatment. These measured biometric signalsallow a technician or health care provider to determine in real-time thelevel of pain and anxiety the patient is experiencing, and thus allowsfor near real-time adjustment to the treatment. Further this measuredpatient condition provides the Virtual Reality Medical ApplicationSystem of the present invention the unique ability to adjust the virtualreality environment to provide the patient with an increased ordecreased level of interaction to make the treatment more successful,and the patient experience more positive.

The technician's workstation 336 allows the technician or health careprovider to view that a patient is seeing, and may also provide for abidirectional communication link between the technician and the patient.Also, the technician workstation 336 may allow a patient to view certainprocedure-specific data, such as images from a surgical biopsy,colonoscopy, etc. thus facilitating real-time consent from a patientduring a procedure. This is of a particular benefit when the inabilityto secure consent from a patient during a procedure could result in aprocedure being terminated prematurely, repeat procedure, or redundantprocedures to accomplish tasks that could have been easily completedwere consent available. Technician workstation 336 may also providereal-time data to a treatment technician via channel 340 to a patientdisplay 342.

Software System Description

FIG. 4 is a system level block diagram of the software of the VirtualReality Medical Application System of the present invention generallydesignated 400. Software block diagram 400 includes a module subsystem402 which includes the user interface module, artificial intelligenceused for character behavior, a rules engine for determining avatarmovement, an achievement module used to track avatar actions, andvarious animations, sound and communication functions. Engine subsystem404 provides rendering, animation, sound, input processing and physicssimulations to provide accurate virtual reality images and physicalbehavior.

A head mounted display (HMD) interface 406 interfaces the display to thegame module subsystem, and supports the rendering of the graphics in thepatient worn virtual reality display. Also, the HMD interface 406provides motion sensing data back to the motion detection interface 410,as well as may provide the patient with a live video feed from thetreatment procedure, facilitating information sharing or consent asmentioned above. To facilitate this communication, communicationinterface 408 interconnects various modules and VOIP services to thegame module system 402.

Motion detection interface 410 receives motion data from various imagingsources 412 and connects the motion detection sensors to the game modulesubsystem. In a preferred embodiment, module 410 utilizes an adaptordesign to accept multiple sensors that can simultaneously orcooperatively monitor the patient position and movement.

Biometric interface 414 connects the various biofeedback sensor inputsto the game module subsystem 402, and utilizes an adapter designed sothat the various biometric sensors may be selectively or simultaneouslymonitored, and can receive information from a biometric data display416.

A technician station 418 includes a technician console 420 and atechnician display 422 that allows bidirectional communication betweenthe technician and the patient, and allows the technician to set motionand treatment parameters and monitor those parameters during thetreatment period, and also allows the technician to monitor the patientbiofeedback data. The technician station 418 is in communication withthe game module subsystem 402 via technician console interface 424.

Operation of the Invention

FIG. 5 is an exemplary flow chart, generally designated 500, anddepicting the method of the pre-treatment simulation of the treatmentusing the Virtual Reality Medical Application System of the presentinvention. Method 500 starts in step 502, and begins with theacquisition of patient identification data, including the photograph ofthe patient and sampling of voice data in step 504. In step 506, thispatient identification data is stored in the patient database.

The physician will decide if biometric sensors will be used and if so,which ones will be used, also the allowable deviations from sensors. Instep 508, the patient's baseline biometric data is acquired, such as theskin conductivity, EKG, EEG, heart rate, respiration rate, bodytemperature and oxygen saturation. Once the baseline biometric data isacquired, it is stored in step 510 in the patient database for laterretrieval. Additionally, limits on the biometric data, such as normalranges and range limitations for detecting unsafe conditions, isdetermined in step 512.

In addition to the biometric baseline data, a patient's physicalpositioning data is acquired in step 514. In treatments where patientposition is critical, the treating physician has the opportunity toaccurately and precisely position the patient for his or her treatmentin the simulation environment thus making certain that the position ofthe patient is optimum for the best treatment outcome possible.

A typical simulation would occur one or two weeks before the radiationtreatment, and would include the patient undergoing a treatment planningprocess. During this simulation, the physician will decide on a certaintreatment position (called, “reference position”) and at that time, a CT(Computed Tomography) is done (sometimes with MRI and/or PET) to planthe radiation treatment (# of beam and directions, # of treatments,outlining the tumor, etc.).

Once this optimum reference position is achieved, the patient positiondata is stored in step 516 for later retrieval. Following the successfulpositioning during the simulation, the patient positional limitationsare determined and stored in step 518. This provides for position limitsetting based on the particular treatment to be performed. For instance,in treatments to the abdomen, such as radiation of cervix or ovariancancer, the precise position of the lower abdomen is critical, whereasthe position of the patient's foot is less critical. In suchcircumstances, the positional limitations on the abdominal position maybe very small, but the positional limitations on the patient's foot maybe relaxed.

Once all data from the simulation has been gathered and stored, method500 returns to its calling procedure in step 520.

FIG. 6 is a flow chart generally designated 530 and representing anexemplary operation of the Virtual Reality Medical Application System ofthe present invention 100. Method 530 includes the preliminary stepsexecuted prior to beginning treatment, begins in start step 532 andincludes initializing the virtual reality system in step 534. Once thepatient identification is verified by acquiring facial image and voicedata in step 536, the patient profile is retrieved from the patientdatabase in step 538. The patient's current biometric data is acquiredin step 540, and the patient biometric and positional limits areretrieved in step 543.

Prior to treatment initiating, the patient's current positional data isacquired in step 544, and compared to the previously stored referenceposition data in step 546. If necessary, the patient or operator makesan adjustment to the patient's position to match the reference positiondata in step 548.

Once positions are matched, the patient is ready to start treatment instep 550. Once ready, the patient biometric and positional data ismonitored starting in step 552, and the patient's virtual realityprotocol is started in step 554. Once fully immersed in the virtualreality environment, the treatment of the patient begins in step 556.

The patient's positional data is measured in step 558 and compared tothe set limitations. If the patient's position is within the set limitsas determined in step 560, the treatment is not completed, as determinedin step 562, flow chart 530 returns to step 558 for continuedmonitoring. If the treatment is completed as determined in step 562, thetreatment ends in step 564.

If the patient position is not within the set limits as determined instep 560, the magnitude of the deviation is determined in step 568. Ifthe deviation is small, the patient is alerted to make an adjustment instep 570, and control returns to step 558 for continued monitoring andtreatment. However, if the deviation is not small, then the treatment ispaused in step 572 until the patient makes a proper position adjustment,and treatment resumes in step 574 and control returns to step 558 forcontinued monitoring and treatment.

Referring now to FIG. 7, a flow chart representing the steps for theacquisition of baseline patient biometric data of FIG. 5 is shown andgenerally designated 600. Flow chart 600 begins with the acquisition ofbaseline patient biometric data in step 602. The reference patient datais accessed in step 604, and that data is stored locally in step 606.

The patient's starting biometrics are measured in step 608, and mayinclude the measurement of 3-D body position, skin conductivity, EKG,EMG, EEG, heart rate, respiration rate, body temperature and oxygenlevel in the blood, and any other biometric data which may be useful forthe procedure being performed.

The patient's reference position is stored in step 610, and the patientstarting biometric data is stored in step 612. Once complete, thecontrol returns in step 614 to the calling procedure.

Referring to FIG. 8, a flow chart representing the steps for theacquisition of baseline physical positioning data of FIG. 6 is generallydesignated 620. Method 620 begins with the acquisition of currentpatient physical positioning data in step 622. The patient referencedata is retrieved in step 624, and the reference data is stored locallyin step 626.

The patient is positioned in his or her proper reference position instep 630, and verified in step 632. If the patient is not in the properreference position, feedback is provided to the patient and technicianin step 634, and the patient position is then verified in step 628. Onceproper positioning is achieved as determined in step 632, controlreturns in step 636 to the calling procedure.

FIG. 9 is an exemplary flow chart, generally designated 700, andrepresenting the beginning of treatment of FIG. 6 without using patientcontroller input. Method 700 begins in step 702 with patientidentification verification using optical, facial and voice recognition.If identity confirmation fails in step 704, the method ends in step 706.However, if identity is confirmed, treatment begins in step 708.

Treatment begins with the activation of the treatment apparatus in step710, and patient biometric data is monitored in step 712, and comparedto patient biometric data tolerances in step 714. If the treatment iscomplete as determined in step 716, the treatment ends in step 736. Ifthe treatment is not complete, the treatment continues in step 718 andthe patient's physical data is monitored in step 720. If the patientphysical data is within tolerance in step 722, treatment continues withreturn to step 712. If the patient physical data is not withintolerances, the physical data is analyzed in step 724 to determine if itis within a safe range. If so, the patient is provided feedback in step732 to facilitate position self-correction, and the technician isalerted in step 734 of the position violation, and method 700 returns tostep 720 for continued monitoring.

If the physical data is outside the safe range as determined in step724, the technician is alerted of the safety issue in step 726, thetreatment apparatus in interrupted in step 728, and the treatment endsin step 730.

If it is determined in step 714 that the patient biometric data is notwithin the preset tolerances, it is determined in step 738 whether thebiometric data is outside the safety range. If safe, the technician isalerted in step 740 of the biometric error, and treatment is adjusted instep 742, and the virtual reality application may increase patient focusin step 746, or the virtual reality application may adjust the programor game to return the patient biometric readings t within range in step748. Once these corrective steps have been taken, method 700 returns tostep 712 for continued monitoring.

If the biometric data as measured is unsafe as determined in step 738,the technician is alerted in step 750 and the treatment apparatus isinterrupted in step 752 and the treatment ends in step 754.

In a preferred embodiment of the method of the present invention, on theday of treatment, a typical treatment process is as follows:

a) Patient is set up on the treatment table;

b) imaging scan the patient's face and record voice to verify it is thecorrect patient. This “Time-out” is the requirement for all treatmentcenters to ensure proper patient identification and treatment;

c) Patient puts on the virtual reality goggles and headset, and thesensors are attached to the patient;

d) imaging scan and assist patient/technician to make adjustment sopatient is in the reference position;

e) This current position is “captured” or recorded together withbiometric sensor data;

f) At this time, the virtual reality game is turned on and operator willleave the treatment room and go to the treatment console;

h) Operator will check with the position monitor and biometric monitorto see they are within limit (“Green light”);

i) Operator will turn on the treatment machine and informed patient viaheadset;

j) Position and biometric sensor monitor are continuously feeding signaland display via monitor:

-   -   “Green Light” (position is good);    -   “Yellow Light” (small deviation can be corrected by patient,        treatment can be continued, and technician is stand by to pause        treatment if necessary; and    -   “Red Light” (Major deviation—treatment must be paused        automatically. Adjustment must be made by patients first, then        technician if needed before treatment can be resumed;

k) If respiration gating feature is used, one more step is added to havepatient holding their breath to certain level and go by Green, Yellow,Red light system; and

l) Treatment completed.

What has been described here are exemplary methods capable using theVirtual Reality Medical Application System of the present invention. Itis to be appreciated that other variations of these methods may beachieved using the present invention, and no limitations as to thesemethods are intended by the few exemplars presented herein.

Referring now to FIG. 10, a flow chart generally designated 800representing the beginning of treatment of FIG. 6 using patientcontroller input, and includes the verification of patient identity instep 802. If the identity cannot be confirmed in step 804, the treatmentends in step 806. If identity is confirmed, treatment begins in step810, and the virtual reality system is initiated in step 812, and thepatient virtual reality program is selected in step 814.

The patient baseline biometric data is input in step 816, and thebiometric data range for the treatment is established in step 818. Thebaseline physical data is input in step 820, and the physical data rangeis established in step 822. The virtual reality program begins in step824 followed by the activation of the treatment apparatus in step 826.

Biometric data is monitored in step 826, and physical data is monitoredin step 830. If data is within range as determined in step 832, thevirtual reality program continues in step 834 to receive input from thepatient in step 836, and to process the patient input in step 838. Asthe virtual reality continues, the program adjusts in response topatient input in step 840, and adjusts in response to biometric datareadings in step 842, and adjusts in response to physical data readingsin step 844. If treatment is complete as determined in step 846,treatment ends in step 848; otherwise, control returns to step 828 tocontinue monitoring biometric data.

If the data measured is out of range as determined in step 832, the datadeviation is determined in step 850. If the deviation is small, a yellowalert is provided to the patient in step 852, and the virtual realityprogram adjusts to address the range violation to provide the patientwith self-correction guidance. If the data deviation is not small asdetermined in step 850, a red alert is given to the patient in step 854,and the treatment is paused in step 856, and the virtual reality programis adjusted to address the range violation in step 858.

Once the virtual reality program has been adjusted, the technician isalerted of the range violation in step 860, and the out-of-range data isrechecked in step 862. Again, in step 832, it is determined if the datais out of range, and the method continues.

Referring now to FIG. 11, an exemplary flow chart is generallydesignated 900 and depicts operation of the game interface of theVirtual Reality Medical Application System of the present invention.Method 900 begins in step 902, and a language is selected in step 904.This language would correspond to the patient's most familiar language,thus providing a sense of comfort to the patient during the treatment.Once language is selected, the game title is displayed in step 906. Aswill be shown below, this game title may correspond to virtually anygame which incorporates patient stabilization features, and is ageappropriate for the patient being treated.

A patient avatar is selected in step 908, and the introduction isdisplayed to the patient in step 910, and instructions are provided instep 912. If the patient does not understand as determined in step 914,then addition instruction is given in step 916. If the patientunderstood the instructions, a tutorial is provided in step 918. If thepatient does not adequately understand or complete the tutorial,addition tutorial is provided in step 922. Once the patient completesthe tutorial successfully as determined in step 920, step 924 displaysthe “cut scene” to transport the patient avatar to the various gameenvironments as determined in step 926, such as determined by thepatient's prior game history or patient age, for example.

The patient avatar may be sent to any number of levels or worlds, suchas level 1 928, level 2, 930, level 3, 932, or level 4, 934. Once on theparticular level or world, the patient's reward screed is displayed instep 936. If the game is over, as determined in step 940, the patientgame data is recorded in step 942, and the game ends in step 944. if thegame is not over, control returns to step 926 and the game continues asthe patient avatar navigates through the various levels, various gameworlds, and successfully completes each challenge within the game.Preferably, the game duration would correspond to the duration of thepatient's treatment such that the attention of the patient is drawnfully into the virtual reality world.

Exemplary Game and User Interlace

A number of exemplary game interfaces are presented herein. Forinstance, FIG. 12 is an exemplary view of a simple virtual reality game.A series of the game display images 1000 are shown as presented in thehead mounted display during the operation of the Virtual Reality MedicalApplication System of the present invention. These video progressionsinclude an image 1002 filled with environmental surroundings such astrees and shrubs 1004 and 1006, and local animals 1008. A child avatar1010 is shown playing in a park. A score panel 1012 keeps track of thepatient's game score. In subsequent frames, the avatar 1010 can movethroughout the display as shown by arrow 1016, and can advance aroundand through the surroundings, such as behind rock 1018. In a simplifiedexample, patient avatar 1010 is being sought out by a friendly puppy1022 which traverses through the display image in path 1024 in friendlypursuit of the patient avatar that hides in direction 1026 behind rock1018. Patient feedback can be used to make certain that no movementoccurs for fear of alerting the puppy 1022 of the avatar's location.Scoring points can be achieved by the successful avoidance of theexuberant puppy 1022. In circumstances corresponding to a criticaltreatment period requiring the patient to remain still, a high scoringsegment 1030 may be used which includes the puppy continuallyapproaching the avatar which ducks in direction 1032 behind the rock toavoid detection.

This simple example of the Virtual Reality Medical Application System ofthe present invention provides for the virtual reality system to providephysical feedback to the patient during treatment to maintain properpositional placement. Moreover, the biometric feedback data from thepatient monitoring can be incorporated into the virtual reality systemto change the avatar environment, increase or decrease patient focus, inorder to increase focus and decrease stress on the patient duringtreatment.

Game Description for FIGS. 13 Through 21

FIGS. 13 through 22 depict an alternative sequence of a game displayedin the Virtual Reality Medical Application System of the presentinvention, which is an animated story in which a patient-linked avatarhelps a dinosaur to retrieve its eggs from an egg-stealing robot byriding a flying skateboard throughout a virtual world to retrieve thestolen eggs, and requires the avatar to remain motionless during flightto avoid losing the trail of the stolen eggs, or dropping any retrievedeggs, thus returning all of the stolen eggs safely to the gratefuldinosaur.

An exemplary title page is shown in FIG. 13, and generally designated1020. FIG. 13 includes an exemplary title page including a game name,and representative game graphics to introduce the patient to the gameenvironment Specifically, display 1020 depicts an animated world, fullof vibrant colors and friendly detail which will appeal to the age groupfor the patient being treated. Understandably, the genre of the virtualreality environment, and the particular details of the game presented inthat environment will change depending on the age of the patient, andthe patient's ability to understand and interact with more complicatedgames. Indeed, the present invention contemplates generally utilizingvirtual reality technology with attractive computer graphics to make agame which can bring kid patients into a virtual world and make immersein it and stay still for no less than 30 minutes when they're having theradiation treatment.

Exemplary features of a virtual reality environment of an embodiment ofthe present invention include:

-   -   Kid patients, ages from 3-10 (older kids may need to be        considered also)    -   Target platform: Personal Computer (PC) or Laptop    -   Display hardware: Oculus Rift II or similar goggle    -   Input device: N/A    -   Extra device: monitor camera    -   Game engine: Unity 3D

In the exemplary game depicted in FIGS. 13 through 21, Dr. Patel is acrazy almighty scientist and dinosaur maniac. His ambition is to build adinosaur park for children on the earth to represent the fantastic eraof dinosaur. By his research, he found there's a planet (D-Planet) inthe universe which has an ecological environment similar to the one theearth used to have back to millions years ago and has dinosaurs activeon it. Thus he wants to get the kid (player) transported by a spaceshipand use the “Aero Skate Board” he created to collect different types ofdinosaur eggs by all mean. There are multiple isolated islands onD-Planet. On each island lives a certain type of dinosaur; variousislands may be provided, each with its only variety of dinosaurs to besaved.

in order to help Dr. Patel to build a dinosaur park on the earth, thechild patient avatar is chosen to be transported to another planet named“D-Planet” which has an ecological environment similar to the one theearth used to have back to millions years ago. There're multiple islandson “D-Planet” and on each island lives a type of dinosaur. The ultimatetask for the chosen child avatar is to utilize a special equipment named“Aero Skate Board” and by all means to search and collect eggs ofdifferent types of dinosaurs on different islands and bring them allback to the earth.

Because one target audience are 3-10 years old child patients, it'simportant to create a virtual world that can let them fully immerse infor 30 minutes, so the art style in general has to be cute, cartoonishwith colorful and bright palettes. Here below are some examples of lookand feel of a preferred embodiment of the present invention depictingcharacters, dinosaurs, items and environments (level)

For instance, if the child patient is a boy, then the game system willassign this boy avatar to him as his virtual identity. He will play therole as the little adventurer chosen by Dr. Patel and accomplish hismission of dinosaur eggs collecting on D-Planet. Alternatively, if thechild patient is a girl, then the game system will assign this girlavatar to her as her virtual identity. She will play the role as thelithe adventurer chosen by Dr. Patel and accomplish her mission ofdinosaur eggs collecting on D-Planet.

FIG. 14 is an exemplar of various game avatar characters. For instance,eccentric doctor Patel 1022, a dinosaur 1024, a girl patient avatar1026, and a boy patient avatar 1028 that can be used in the game of theVirtual Reality Medical Application System of the present invention toaccomplish the virtual reality environments that capture the childpatient's imagination during his or her treatment.

FIG. 15 is an exemplary avatar selection screen in which an eccentricdoctor selects a boy or girl (patient) avatar, such as by the patientrotating his or her head in the direction of the desired avatar, such asturning the head right to select the girl avatar, or left to select theboy avatar, with the selection being advanced to the next game screen;

Dr. Patel walks into the screen from right (screen 1032).

Makes self introduction.

Inside Dr. Patel's Lab (a rounded hall)

A big screen hangs from the ceiling of the lab.

Dr. Patel asks patient to identify gender (Screen 1034).

Boy turns head to left (Screen 1036)/Girl turns to right (screen 1038).

Grey area can only be seen when patient turns his/her head whenidentified.

Dr. Patel walks towards boy (screen 1036).

Boy activated when Dr, reaches him.

Dr. leads boy return to center of screen (Screen 1040).

Patient needs to turn his head back.

Camera moves towards to show the big screen.

Dr. Patel starts to introduce the mission.

D-Planet map shows on the screen (turning slowly).

Boy faces towards the screen and listens.

Using the virtual reality environment, a variety of gameplay options areavailable, and the specific examples presented herein are not to beconstrued as any limitation of the present invention.

Gameplay Option 1

In each level, the chosen child avatar must hide behind a certain objectand keep still in the game scene for a certain period of time (60 sec-90sec) when the Egg Collector is searching and collecting dinosaur eggs.If the patient moves any part of the body during the period of time thenthe patient avatar will correspondingly move in the virtual realityenvironment, and the dinosaur parents will be aware and go to protecteggs and destroy the Egg Collector.

The Egg Collector has a countdown clock on its surface to show timecountdown. If the child patient moves any part of the body, the movementwill be detected by the motion monitoring system and child patient'savatar in the game will move correspondingly.

The Egg Collector will also stop collecting eggs for a few seconds. Ifthe child patient moves 3 times during the period of time, the eggcollection mission failed and the patient avatar will need to re-do thesame mission.

The patient avatar will get rewarded with a bronze egg or saver egg orgold egg based on the behavior in the finished level. Egg reward of eachlevel has different/unique design, and given that many treatmentsrequire multiple sessions, it is conceivable that a patient will be ableto collect many different rewards over the course of treatment.

Gameplay Option 2

Mother dinosaur lost 5 eggs in each level and it's hard for her to findthem back because she has other eggs need to be taken care of. Thechosen kid volunteered to find all 5 eggs back.

Riding a hi-tech “aero skateboard” and carrying an egg-protector on theback, the avatar starts the journey in the level to look for lost eggs.He/She needs to standing on the skateboard by keeping balance (in orderto make it flying smooth and stable). If the child patient moves anypart of the body, the movement will be detected by the monitor camera,body position, and patient's avatar in the game will movecorrespondingly and this will make the skate board unbalanced.

If the child patient moves 3 times during the period of time, 1^(st)time caution light on the tail of skateboard will turn into orange andflickering; 2^(rd) time caution light on the tail of skateboard willturn into orange and flickering rapidly; 3^(rd) time mission failed.

The patient avatar will receive rewards from the mother dinosaur with anegg if the level is completed. Each egg reward of each level hasdifferent/unique design (to indicate a certain type of dinosaur).

FIG. 16 is an exemplary display 1050 of a game within the VirtualReality Medical Application System of the present invention depictingthe patient avatar on a flying skateboard and traveling through avirtual forest in pursuit of lost dinosaur eggs. This is an example of apreferred embodiment of the virtual reality environment.

Referring now to FIG. 17, an exemplary map of a game within the VirtualReality Medical Application System of the present invention is shown andgenerally designated 1060. A representative forest 1062 with colorfultopography, and the advancement of the patient avatar from an origin1064 to an end point 1080 is designed to allow the avatar 1068 to gatherthe stolen eggs 1070, 1072, 1074, 1076, and 1078 from the forest. Inthis version, the patient avatar will ride the skateboard and start eggcollecting from point green 1064 and end up at point red 1078 by theroute line showing in the image on the left. In this preferredembodiment, the route is set up by program and it cannot be changed.Other embodiments feature a dynamic route determined by the randomplacement of the eggs throughout the forest 1062, such that eachexperience for the patient will vary.

The estimated time of finishing the level will be around 4˜5 to 10minutes. If the child patient moves his or her body during the game andcause the dropping or loss of any eggs, then the level will restart fromthe very beginning.

FIG. 18 is an exemplary display 1100 of the game showing the patientavatar 1102, the dinosaurs 1104, a number of eggs 1106, the egg-stealingrobot 1108, with the patient avatar 1102 hiding from the robot 1108 inorder to avoid detection to protect the dinosaur eggs.

A number of game statistics may be displayed on image 1100, includingegg value 1110, eggs collected 1114 and time counter 1116. This dataprovides the patient with information related to his or her currentscore, the duration of the game, and the progress through the gamesession.

FIG. 19 is an exemplary display 1120 of the game showing the patientavatar 1128 riding a flying skateboard 1130 through the virtual forestenvironment in search for stolen dinosaur eggs 1126, and providing apatient with a motion status indicator light 1132 on the skateboardcorresponding to patient movement measurement feedback to the patient,such as green for good, yellow for minimal patient motion detected, andred for too much motion detected. An exemplary map 1134 identifies wherethe patient avatar 1128 is within the virtual forest path. Also depictedin this view is the number of eggs collected 1122, and a dinosaur 1124to be avoided.

The image on the bottom shows the scene the child patient will see inthe virtual reality goggles when the game starts. For instance, fivedinosaur eggs can be seen on the way (determined by the route set up)and the skateboard will lead the way to it automatically. Once adinosaur egg is collected, an egg-figured space on the top left screenwill be filled with an egg 1122 icon correspondingly.

At the conclusion of the game session, a representative game end screenis shown in FIGS. 20 and 21. These screens report the patient's successin collecting all of the stolen eggs from the virtual forest The FIG. 20screen 1140 shows after current level be completed successfully.“Result” shows the number of dinosaur eggs 1142 that were saved in thecompleted level. “Reward” shows the reward egg (the certain type of egg)player gets from dinosaur mother. Alternatively, the FIG. 21 screen 1150is representative of the patients' failure to collecting all of thestolen eggs 1152 from the virtual forest. This screen shows after theplayer failed to completed current level. System will automaticallyswitch back to current level and let player play it again. Neither“Result” nor “Reward” shows on this screen.

Referring to FIG. 22, a representative display of the array of IMU forbody motion detection is shown and generally designated 1200. Each IMU1202 includes the detectors necessary to sense the inertial movement ofthe patient. This data is relayed wirelessly to the system of thepresent invention in order to communicate real-time position, anddynamic changes in position, to the technician or health care providerin order to accomplish the treatment protocol.

FIG. 23 is a representative display of the IMU for the respiratorygating application showing the placement of a single IMU on the abdomenof a patient. From this Figure, it can be appreciated that any varietyand placement of IMU devices are fully contemplated in this invention,and can be configured to provide the technician or medical care providerthe positional information required to provide safe, effective andrepeat procedures.

Another embodiment of the invention is the respiratory gatingapplication. FIG. 24 is an exemplary display of the game showing how thegame can assist the patient in holding their breath and breathingproperly for respiratory gating applications. The application startswith a screen option where the technician has to record the maximum andthe minimum breathing value of the patient. The IMU motion sensor isstrapped on the chest of the patient where the maximum breath-in andbreathe-out can be recorded The patient is asked to breath in at amaximum point (maximum inspiration) and the technician records thevalue. Similarly the patient is asked to breathe out maximum breathvolume to record the breathe-out value (maximum expiration). Thesevalues are stored in the database which will be used during the game.

As shown in the lower portion of FIG. 24, the position of the IMU duringa patients breathing is recorded and displayed for the technician ormedical care provider to gauge the patient's breathing, and to verifythat the patient breathing pattern is gated according to the gamestimulus, and corresponding to the treatment being provided to thepatient.

In a preferred embodiment, the game incorporated into the presentinvention is about a girl on a skateboard who travels inside a sci-fitunnel. When the patient breathes-in, the skateboard moves up and whenhe/she breathes out the skateboard moves down. The values from thesensors are linked to the movement of the skateboard. The patient shouldnot breathe more than the value recorded as the skateboard might hit thetop of the tunnel and slows down the skateboard or he/she should notbreathe out more since the skateboard might hit the floor and slows itdown.

The ultimate goal of the game is to avoid hitting the tunnel and cover amaximum distance within the prescribed time of 45 seconds to 1 minute.Game points will be awarded on how long the breath is held+m inimalchest move up and down and the speed. Once the patient treatment iscomplete, the virtual reality environment returns the patient to thereal world environment gradually in order to provide a smooth transitionfrom the virtual reality environment and minimize stress upon thepatient.

The Virtual Reality Medical Application System presently describedherein is capable of obtaining the objects of the present invention. Theparticular preferred embodiments that have been presented herein aremerely exemplary of preferred embodiments, but various alternativecombinations of the features and components of the present invention maybe constructed and are fully contemplated in the present invention.

What is claimed is:
 1. A method for monitoring and controlling bodymovements of a patient during a radiation procedure for increasedefficacy, comprising: providing a radiation procedure apparatus with aradiation procedure area for performing the radiation procedure;providing a head mounted display configured to detect head movement;providing an array of motion detector sensors configured to detect bodymovement; providing a plurality of biofeedback sensors configured todetect biometric data; providing a virtual reality console system,wherein the virtual reality console system is in communication with thehead mounted display, the array of motion detector sensors, and theplurality of biofeedback sensors; providing a radiation procedureapparatus control system, wherein the radiation procedure apparatuscontrol system is in communication with the virtual reality consolesystem and the radiation procedure apparatus; positioning the patient inthe radiation procedure area of the radiation procedure apparatusutilized to perform the radiation procedure; positioning the headmounted display on the patient; configuring the array of motion detectorsensors to detect body movement of the patient; retrieving a biometricdata limits and a position data limits for the patient; tracking acurrent patient position of the patient in the radiation procedure areautilizing the head mounted display to detect head movement and the arrayof motion detector sensors to detect body movement; guiding the patientinto a reference position in the radiation procedure area for theradiation procedure; rendering a patient specific 3-D game by thevirtual reality console system; displaying the patient specific 3-D gameto the patient through the head mounted display; displaying the currentpatient position as a virtual avatar in the 3-D game, wherein thevirtual avatar is controlled by the current patient position; renderingsound to the patient with a noise-cancellation headphone and amicrophone system in the head mounted display to provide a 3-waycommunication between the patient, the radiation procedure apparatuscontrol system, and the virtual reality console system; performing theradiation procedure; sensing current biometric data of the patient andcomparing the current biometric data with the biometric data limitsduring the radiation procedure; communicating the current biometric dataof the patient to a healthcare provider through the radiation procedureapparatus control system; displaying a current avatar position of theavatar in the patient specific 3-D game to keep the patient in apredetermined position to improve efficacy of the radiation procedure;and providing cues through patient specific 3-D game with the avatar tothe patient to adjust the patient's position to the reference positionto improve efficacy of the radiation procedure, said cues comprisingfeedback via the avatar, pausing the patient specific 3-D game, commandsfrom a game character, negative reinforcement in the patient specific3-D game, and positive reinforcement in the patient specific 3-D game.2. The method of claim 1, comprising generating a virtual simulation ofreal life scenery.
 3. The method of claim 1, comprising rendering imagesof a procedure that the healthcare provider wants to share with thepatient.
 4. The method of claim 1, comprising detecting one or more of:blood pressure, heart rate, EEG, and EKG.
 5. The method of claim 1,comprising performing gesture recognition, facial recognition and voicerecognition with the virtual reality console system.
 6. The method ofclaim 1, comprising providing treatment for one of: radiation therapy,brachytherapy, Computed Tomotherapy (CT), Positron Emission Tomography(PET), Magnetic Resonance Imaging (MRI), angiography, biopsy andendoscopy.
 7. The method of claim 6, comprising sharing clinicalinformation from medical diagnostic body scan, angiography, or endoscopywith the patient using the multimedia interaction.
 8. The method ofclaim 1, comprising playing a game that negatively reinforces thepatient to not move utilizing game play mechanics.
 9. The method ofclaim 8, comprising providing levels of positive reinforcement to rewardthe patient for remaining still.
 10. The method of claim 1, comprisinga. playing a game wearing the head mounted display; b. provide a viewinto the virtual world where the patient can see the virtual avatar; andc. tracking movement of the patient's body and when the patient moves inthe real world, moving the virtual avatar in the virtual world in realtime.
 11. The method of claim 1, further comprising: determining whethermovements of the patient exceed the position data limits; pausing themedical mission when movements of the patient exceed the position datalimits until the position of the patient is adjusted to the referenceposition.
 12. The method of claim 1, further comprising: sendingfeedback to the patient to hold the patient's breath and maintain apredetermined chest position; sending a signal to allow a radiation beamto turn on while the chest of the patient is in the predetermined chestposition; and timing the radiation beam based on the patient biometricdata.